Bewitched, Bothered, and Bewildering: The Use and Misuse of State SAT and ACT Scores

Twelve years ago, Brian Powell and Lala Carr Steelman analyzed state SAT scores in a landmark article in the Harvard Educational Review. At the time, politicians and the media, among others, had been using raw state SAT scores to make inferences about the relative quality of education among the U.S. states. Powell and Steelman, however, found that more than 80 percent of the variation in average state SAT scores could be attributed to the percentage of students in a state taking the test in other words, in states where the percentage of students taking the SAT was low, state SAT averages tended to be high because that test-taking population included a high proportion of high-achieving students, Bewitched, Bothered, and Bewildering: The Use and Misuse... http://www.edreview.org/harvard96/1996/sp96/p96powel.htm 2 of 38 2/9/2007 9:11 AM and vice-versa. Since the percentage of students taking the SAT was not necessarily linked to the quality of education in a given state, Powell and Steelman cautioned against using unadjusted state SAT averages to evaluate educational quality. In this article, Powell and Steelman revisit the subject of state SAT scores, providing an update on how state SAT scores continue to be used and misused in public deliberation over the last decade, reanalyzing interstate variation in SAT scores using contemporary data, and extending their analysis to investigate variation among state ACT scores. Powell and Steelman conclude by reaffirming their earlier position that state rankings based on SAT scores change dramatically once they have been adjusted for factors such as the participation rate or the class rank of the student test-taking population. In addition, despite the claims of some researchers and policymakers that money does not make much difference in terms of student achievement, Powell and Steelman find that public expenditures are positively related to state SAT and ACT performance. (pp. 27-59) Over a decade ago, we (Powell & Steelman, 1984) were the first to publish a major analysis of factors linked to interstate variation in Scholastic Assessment (formerly Aptitude) Test (SAT) performance.1 Using previously unreleased and unexamined state data on SAT scores provided by the College Board, we analyzed state and regional differences in SAT scores as a function of several compositional and structural variables. Among our major conclusions were: 1. The preponderance of variation over 80 percent in state SAT averages is attributable to the participation rate (i.e., the percent of students in each state taking the SAT). 2. Other features of the test-taking population, namely sex composition, racial composition, and median income, also explain a large portion of state differences. 3. When these compositional factors are taken into account, the relative rankings of state performance are markedly altered. 4. Upon considering compositional factors, per capita expenditures appear to influence considerably state SAT scores. Our current article has several objectives. First, we rearticulate our original position that Bewitched, Bothered, and Bewildering: The Use and Misuse... http://www.edreview.org/harvard96/1996/sp96/p96powel.htm 3 of 38 2/9/2007 9:11 AM unadjusted SAT rankings of states should not be used to gauge the quality of state educational systems. Second, we probe how state SAT rankings have been employed in the interim (i.e., since 1984) to steer social policy and stoke political fires. Third, we reanalyze the issue using contemporary state SAT data and other refinements. Fourth, we analyze data examining state rankings with respect to performance on another widely used standardized test, the American College Test (ACT).2